1 



MOVING OBJECT DETECTOR AND IMAGE MONITORING SYSTEM 

BACKGROUND OF THE INVENTION 
Field of the Invention 
5 The present invention relates to a moving object detector 

for detecting one or more moving objects on a screen by using 
an encoding parameter and an image monitoring system provided 
with the moving object detector. 
Description of the Prior Art 

10 In a prior art image monitoring system intended for crime 

prevention or disaster prevention using a surveillance camera^ 
an enormous load is put on an observer because the observer should 
be stationed full time so as to check a monitor or the like at 
anytime, an enormous load is also put on a transmission path 

15 because of constant flow of coded streams through the 
transmission path, and a large amount of storage used for storing 
coded streams is needed because the coded streams are stored 
continuously. Therefore, in the prior art image monitoring 
system a moving object detector for automatically detecting one 

20 or more moving obj ects is adopted . Japanese patent application 
publication No. 9-2524 67 discloses an example as shown in Fig. 
11 of such a moving object detector. 

The prior art moving object detector shown in Fig. 11 is 
disposed on a side of a decoding apparatus, and is provided with 

25 a motion vector extraction unit 1102 for extracting a motion 
vector indicating an amount and direction of movement for each 
macroblock from a coded stream applied to the decoding apparatus, 
and a moving object determination unit 1103 for performing 
statistical processing on the amount and direction of the motion 

30 vector for each macroblock, which are extracted by the motion 
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vector extraction unit 1102, and on positions and number of 
macroblocks associated with the motion vector so as to detect 
and keep track of one or more moving objects. 

However, a problem with the above-mentioned prior art 
5 moving object detector is that since a motion vector is not 
necessarily extracted from each macroblock including a moving 
object according to an encoding mode determined for each 
macroblock during encoding of a moving image, it is impossible 
to accurately detect one or more moving objects by using only 

10 motion vectors . Particularly, since in anMPEG-4 video encoding 
system the moving image is encoded at a low bit rate and with 
a low resolution inmost cases, there may be cases where no motion 
vector is extracted from a macroblock including a moving ob j ect . 

Furthermore, a problem with the above-mentioned prior art 

15 image monitoring system is that since coded streams are 
transmitted and stored even when no moving object exists in the 
monitoring site on the assumption that the observer should be 
stationed full time so as to check a monitor or the like at anytime 
and the moving image is continuously stored, an enormous load 

20 is put on the transmission path and a large amount of storage 
used for storing the coded streams is needed. 

Another problem is that when a moving object is detected 
from a stored coded stream, it takes much time to search through 
the coded stream for data associated with the moving object. 

25 

SXMMARY OF THE INVENTION 
The present invention is proposed to solve the 
above-mentioned problems, and it is therefore an object of the 
present invention to provide a moving object detector that can 
30 accurately detect one or more moving objects by changing a 
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criterion used for detection of moving objects according to an 
encoding mode in which, an encoding unit is placed even when in 
the encoding mode no motion vector is generated, and an image 
monitoring system that employs the moving object detector. 
5 It is another object of the present invention to provide 

a moving object detector that can lighten the load put on a 
transmission path by neither transmitting the coded stream nor 
storing the coded stream when no moving object exists in the 
monitoring site, and an image monitoring system that employs 

10 the moving object detector. 

It is a further object of the present invention to provide 
a moving object detector that, when it is determined when one 
or more moving objects exist in the monitoring site, can 
facilitate searching for a frame at which the one or more moving 

15 ob j ects are detected by storing a coded stream which starts from 
an intra-coded frame together with metadata, and an image 
monitoring system that employs the moving object detector. 

In accordance with an aspect of the present invention, 
there is provided a moving object detector comprising: an 

20 effective macroblock identification unit for identifying one 
or more candidates for a macroblock that can be contained in 
an on-screen moving object as one or more effective macroblocks 
by using an encoding parameter ; and a moving obj ect determination 
unit for determining whether or not each of the one or more 

25 effective macroblocks identified by the effective macroblock 
identification unit is contained in a moving object based on 
a number of effective macroblocks (referred to as a number of 
adjacent effective macroblocks from here on) which are directly 
adjacent to each of the one or more effective macroblocks or 

30 indirectly adjacent to each of the one or more effective 



macroblocks via one or more other effective macroblocks . 

In accordance with another aspect of the present invention, 
the moving object determination unit counts the number of 
adjacent effective macroblocks for each of the one or more 
effective macroblocks identified by the effective macroblock 
identification unit, and, when the number of adjacent effective 
macroblocks exceeds a predetermined threshold value, determines 
that a group of plural effective macroblocks directly or 
indirectly adjacent to one another and including each of the 
one or more effective macroblocks is a moving object. 

In accordance with a further aspect of the present 
invention, the moving object detector further comprises a coded 
stream analyzer for analyzing a coded stream obtained by encoding 
a moving image so as to extract an encoding parameter from the 
coded stream, and for outputting the encoding parameter to the 
effective macroblock identification unit. 

In accordance with another aspect of the present invention, 
the moving object detector further comprises an encoding 
parameter input unit for receiving an encoding parameter 
generated when a moving image is encoded, and for outputting 
the encoding parameter to the effective macroblock 
identification unit. 

In accordance with a further aspect of the present 
invention, the moving obj ect detector further comprises a moving 
obj ect coordinate-storage unit for storing on-screen coordinates 
of each effective macroblock which has been determined as a 
macroblock contained in a moving object by the moving object 
determination unit, a motion vector storage unit for storing 
a motion vector for each effective macroblock which has been 
determined as a macroblock contained in the moving object, and 
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a coordinate calculation unit for calculating a prediction of 
the on-screen coordinates of each effective macroblock in a next 
frame based on the on-screen coordinates of each effective 
macroblock stored in the moving object coordinates storage unit 

5 and the motion vector for each effective macroblock stored in 
the motion vector storage unit . The moving obj ect determination 
unit then determines whether or not each of the one or more 
effective macroblocks identified by the effective macroblock 
identification unit is contained in a moving object by using 

10 the one or more effective macroblocks andmacroblocks each having 
a prediction of its on-screen coordinates calculated by the 
coordinate calculation unit. 

In accordance with another aspect of the present invention, 
the moving object determination unit determines whether or not 

15 each of the one or more effective macroblocks identified by the 
ef f ectivemacroblock identification unit is contained in amoving 
object based on the number of adjacent effective macroblocks 
which are an overlap among a group of plural effective macroblocks 
directly or indirectly adjacent to one another and including 

20 each of the one or more effective macroblocks and the macroblocks 
each having a prediction of its on-screen coordinates calculated 
by the coordinate calculation unit. 

In accordance with a further aspect of the present 
invention, the moving obj ect determinationunit counts the number 

25 of adjacent effec-tive macroblocks which are an overlap among 
a group of plural effective macroblocks directly or indirectly 
adjacent to one another and including each of the one or more 
effective macroblocks and the macroblocks each having a 
prediction of its on-screen coordinates calculated by the 

30 coordinate calculation unit, and then determines whether or not 
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each of the one or more effective macroblocks identified by the 
ef fectivemacroblock identification unit is contained in amoving 
object based on the counted number of adjacent effective 
macroblocks . 

5 In accordance with a further aspect of the present 

invention, the effective macroblock identification unit 
identifies a macroblock as an effective macroblock that can be 
contained in a moving object when an encoding mode of the 
macroblock extracted as the encoding parameter of the macroblock 

10 is an intra-coding mode. 

In accordance with another aspect of the present invention, 
the effective macroblock identification unit identifies a 
macroblock as an effective macroblock that can be contained in 
a moving ob j ect when an encoding mode of the macroblock extracted 

15 as the encoding parameter of the macroblock is a motion 
compensation prediction mode. 

In accordance with a further aspect of the present 
invention, the effective macroblock identification unit 
identifies a macroblock as an effective macroblock that can be 

20 contained in a moving object when a number of DCT coefficients 
of the macroblock extracted as the encoding parameter of the 
macroblock is equal to or greater than a predetermined threshold 
value . 

In accordance with another aspect of the present invention, 
25 the effective macroblock identification unit identifies a 
macroblock as an effective macroblock that can be contained in 
a moving object when a sum of codes of AC components of DCT 
coefficients of the macroblock extracted as the encoding 
parameter of the macroblock is equal to or greater than a 
30 predetermined threshold value. 



In accordance with a further aspect of the present 
invention, when an encoding mode of a macroblock extracted as 
the encoding parameter of the macroblock is amotion compensation 
prediction mode, the effective macroblock identification unit 
calculates a normof amotion vector calculated for the macroblock, 
and identifies the macroblock as an effective macroblock that 
can be contained in a moving object when the calculated norm 
is equal to or greater than a predetermined threshold value. 

In accordance with a further aspect of the present 
invention, there is provided an image monitoring system including 
amoving image decoding unit for receiving a coded stream obtained 
by encoding an image generated with a camera by way of a 
transmi s s ion path and for decoding the coded stream, and a monitor 
for displaying the decoded image obtained by the moving image 
decoding unit, the system comprising: a moving object detector 
including an effective macroblock identification unit for 
identifying one or more candidates for a macroblock that can 
be contained in an on-screen moving ob j ect as one or more effective 
macroblocks by using an encoding parameter, and a moving object 
determination unit for determining whether or not each of the 
one or more effective macroblocks identified by the effective 
macroblock identification unit is contained in a moving object 
based on a number of effective macroblocks which are directly 
adjacent to each of the one or more effective macroblocks or 
indirectly adjacent to each of the one or more effective 
macroblocks via one or more other effective macroblocks. 

In accordance with another aspect of the present invention, 
when the moving obj ect detector determines that any moving ob j ect 
does not exist in a preceding frame but one or more moving obj ects 
exist in a current frame, the moving object detector transmits 
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a signal indicating a request for intra coding to a moving image 
encoding unit that generates the coded stream by way of the 
transmission path and the moving image encoding unit then 
intra-codes following frames of the image when receiving the 
request signal from the moving object detector. 

In accordance with a further aspect of the present 
invention, the moving image encoding unit intra-codes following 
frames of the image and also transmits information such as a 
time and characteristics of the image as metadata by way of 'the 
transmission path when receiving the request signal from the 
moving object detector. 

In accordance with another aspect of the present invention, 
when the moving object detector detects a moving object, the 
moving object detector outputs information such as a time and 
characteristics of the image as metadata. 

In accordance with a further aspect of the present 
invention, there is provided an imagemonitoring system including 
a camera for generating an image of a monitor point, and a moving 
image encoding unit for encoding the image generatedby the camera 
and for outputting the encoded image as a coded stream, the system 
comprising a moving object detector including an effective 
macroblock identification unit for identifying one or more 
candidates for a macroblock that can be contained in an on-screen 
moving object as one or more effective macroblocks by using an 
encoding paramete-r; and a moving object determination unit for 
determining whether or not each of the one or more effective 
macroblocks identified by the effective macroblock 
identification unit is contained in a moving object based on 
a number of effective macroblocks which are directly adjacent 
to each of the one or more effective macroblocks or indirectly 
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adjacent to each of the one or more effective macroblocks via 
one or more other effective macroblocks . When the moving obj ect 
detector detects a moving object, the moving object detector 
transmits a warning signal by way of a transmission path. 
5 In accordance with a further aspect of the present 

invention, only when the moving object detector determines that 
a moving object exists on a screen, the moving image encoding 
unit transmits the coded stream by way of the transmission path. 
In accordance with another aspect of the present invention, 

10 when the moving obj ect detector determines that any moving obj ect 
does not exist in a preceding frame but one or more moving obj ects 
exist in a current frame, the moving image encoding unit 
intra-codes following frames of the image. 

In accordance with a further aspect of the present 

15 invention, only when the moving obj ect detector determines that 
a moving object exists on a screen, the moving image encoding 
unit intra-codes following frames of the image and also transmits 
information such as a time and characteristics of the image as 
metadata by way of the transmission path. 

20 Further objects and advantages of the present invention 

will be apparent from the following description of the preferred 
embodiments of the invention as illustrated in the accompanying 
drawings . 

25 BRIEF DESCRIPTION OF THE DRAWINGS 

Fig . 1 is a diagram showing the structure of a moving obj ect 
detector 101 according to a first embodiment of the present 
invention; 

Fig. 2 is a diagram showing an example of effective 
30 macroblocks identified by an effective macroblock 
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identification unit 103; 

Fig, 3 is a flow chart showing an example of a flow of 

a determination process of determining whether each effective 

macroblock is contained in a moving object, which is performed 

by the moving object detector 101 according to the first 

embodiment of the present inventions- 
Fig. 4 is a block diagram showing the structure of a moving 

object detector 101 according to a variant of the first 

embodiment ; 

Fig . 5 is a block diagram showing the structure of a moving 
object detector 4 01 according to a second embodiment of the 
present inventions- 
Fig. 6 is a diagram showing a relationship between a 
prediction of the on-screen coordinates of effectivemacroblocks 
estimated from corresponding effective macroblocks in an 
immediately preceding frame shown in Fig. 2, and the on-screen 
coordinates of effective macroblocks in a current frame, which 
are identified by an effective macroblock identification units- 
Fig . 7 is a block diagram showing the structure of a moving 
object detector 401 according to a variant of the second 
embodiment of the present inventions- 
Fig. 8 is a block diagram showing the structure of an image 
monitoring system according to a third embodiment of the present 
inventions- 
Fig. 9 is a ]>lock diagram showing the structure of an image 
monitoring system according to a variant of the third embodiment 
of the present inventions- 
Fig. 10 is a block diagram showing the structure of an 
image monitoring system according to a fourth embodiment of the 
present inventions' and 
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Fig. 11 is a block diagram showing the structure of a prior 
art moving object detector. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
Einbodiment 1. 

Fig . 1 is a block diagram showing the structure of a moving 
obj ect detector 101 according to a first embodiment of the present 
invention. In Fig. 1, reference numeral 102 denotes a coded 
stream analyzer that receives a coded stream encodedby an encoder 
(not shown in the figure) , analyzes the input coded stream, and 
extracts encoding parameters for each macroblock from the input 
coded stream, reference numeral 103 denotes an effective 
macroblock identification unit for identifying one or more 
macroblocks each of which becomes a candidate of a macroblock 
that can be contained in a moving object on a screen by using 
one or more encoding parameters extracted by the coded stream 
analyzer 102, and reference numeral 104 denotes a moving object 
determination unit for determining whether or not each effective 
macroblock identified by the effective macroblock 
identification unit 103 is a macroblock contained in a moving 
object. 

In operation, when a coded stream of a moving image is 
input to the moving obj ect detector 101, the coded stream analyzer 
102 performs variable-length decoding on the input coded stream 
so as to extract e-ncoding parameters for each macroblock, such 
as an encoding mode, DCT coefficient values, the amount of codes 
of DCT coefficients, the amount of codes of each macroblock, 
and a motion vector, from the input coded stream. The coded 
stream input to the moving object detector 101 can be the one 
which conforms to any one of all existing moving picture coding 
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methods which use motion compensationpredictions and orthogonal 
transformations, such as ITU-T Recommendation H.2 61 coding 
method, ITU-T Recommendation H.2 63 coding method, the MPEG-1 
video coding method, the MPEG-2 video coding method, and the 
5 MPEG- 4 video coding method, and another moving picture coding 
method that will be newly designed in the future. 

The effective macroblock identification unit 103 
identifies one or more macroblocks each of which can be contained 
in a moving object, as effective macroblocks for the moving obj ect, 

10 by using one or more encoding parameters extracted by the coded 
stream analyzer 102, and outputs the on-screen coordinates of 
each effective macroblock identified. As a means for 
identifying effective macroblocks, there is a means for 
determining that all macroblocks whose encoding mode is intra 

15 coding are candidates for a macroblock that can be contained 
in a moving object because such a macroblock is moving fast and 
therefore the intra coding mode having a small correlation with 
a preceding frame has been selected. Furthermore, when the 
encoding mode of a macroblock is the one, such as the MPEG-4 

20 encoding mode, that provides a motion compensation prediction 
for each 8 by 8 block, the macroblock can be identified as a 
candidate for a macroblock that can be contained in a moving 
object because it is determined that the macroblock is moving 
fast and therefore the encoding mode that provides a motion 

25 compensation prediction for each 8 by 8 block has been selected. 
Alternatively, the effective macroblock identification unit 103 
can calculate a norm (maximum value) of the motion vector for 
each macroblock, and can then identify each macroblock as a 
candidate for a macroblock that can be contained in a moving 

30 object when the norm is equal to or greater than a predetermined 
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threshold value. As an alternative, the effective macroblock 
identification unit 103 can identify each macroblock as a 
candidate for a macroblock that can be contained in a moving 
object when the number of non-zero DCT coefficients is equal 
to or greater than a predetermined threshold value or when the 
sum total of codes of the AC components of the DCT coefficients 
is equal to or greater than a predetermined threshold value. 
As an alternative, the effective macroblock identification unit 
103 can identify each macroblock as a candidate for a macroblock 
that can be contained in a moving object when the amount of codes 
of the DCT coefficients for each macroblock or the amount of 
codes of the encoding parameters for each macroblock is equal 
to or greater than a predetermined threshold value. 
Alternatively, two or more of these identification means can 
be combined. In other words, any technique of determining 
whether or not each macroblock is a candidate for a macroblock 
that can be contained in a moving ob j act using one or more encoding 
parameters is acceptable. 

Fig. 2 is a diagram showing an example of effective 
macroblocks identified by the effective macroblock 
identification unit 103. In the figure, reference nxameral 31 
denotes a screen, and reference numerals 301 to 329 denote 
effective macroblocks, respectively. 

The moving object determination unit 104 determines 
whether or not each effective macroblock is contained in a moving 
object based on information on a position of each effective 
macroblock identified by the effective macroblock 
identification unit 103 according to a flow chart as shown, for 
example, in Fig. 3. Fig. 3 shows an example of the flow of the 
process of determining whether or not each effective macroblock 
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is contained in a moving object, which is performed by the moving 
object determination unit 104. First of all, the moving object 
determination unit 104, in step STl, counts the number of 
effective macroblocks (referred to as a number of adjacent 
effective macroblocks from here on) which are directly adjacent 
to a selected effective macroblock, and further, in step ST2, 
counts the number of adjacent effective macroblocks which are 
indirectly adjacent to the selected effective macroblock via 
another ef f ectivemacroblock, i.e., which are adj acent to another 
effective macroblock already counted in step STl or ST2, and 
which have not been counted yet in steps STl and ST2, and adds 
the counted number in step ST2 to the counted number in step 
STl. 

The moving object determination unit 104 then repeats the 
counting of such adj acent effective macroblocks until it performs 
the counting for all effective macroblocks which are directly 
or indirectly adj acent to the selected eff ectivemacroblock (when 
"Yes" in step ST3, the moving object determination unit 104 
returns to step ST2) . 

Here, in the case of the example shown in Fig. 2, the number 
of adjacent effective macroblocks is 0 for each of the effective 
macroblocks 301 to 303 because there is no effective macroblock 
which is adjacent to each of the effective macroblocks 301 to 
303. In the case of the effective macroblock 304, the number 
of adjacent effec-tive macroblocks is 23 because the effective 
macroblocks 305 to 327 which are directly or indirectly adj acent 
to the effective macroblock 304 are counted in steps STl and 
ST2. In the case of the effective macroblock 328, the number 
of adjacent effective macroblocks is 1 because the effective 
macroblock 329 which is directly adjacent to the effective 
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macroblock 328 is counted in step STl. 

When the moving object determination unit 104 completes 
the counting of the number of adjacent effective macroblocks 
for the selected effective macroblock (i.e., when "No" in step 
5 ST3), the moving object determination unit 104 then, in step 
ST4, compares the sum total of adjacent effective macroblocks 
counted in steps STl and ST2 for the selected effective macroblock 
with a predetermined threshold value, and then determines that 
the region including all counted adjacent effective macroblocks 

10 is contained in the region of a moving object when the sum total 
of adjacent effective macroblocks is equal to or greater than 
the predetermined threshold value (i.e., when "Yes " in step ST4 ) . 
In other words, the moving object determination unit 104 
determines that the region of the group or cluster including 

15 all counted adjacent effective macroblocks is a moving object 
when the sum total of adjacent effective macroblocks is equal 
to or greater than the predetermined threshold value. 

In contrast, when the sum total of adjacent effective 
macroblocks is less than thepredetermined threshold value (i.e., 

20 when "No" in step ST4), the moving object determination unit 
104 determines that the region including all counted adjacent 
effective macroblocks is not contained in the region of any moving 
object. In other words, when the sum total of adj acent effective 
macroblocks is less than the predetermined threshold value, the 

25 moving object determination unit 104 determines that the region 
of the group of one or more adjacent effective macroblocks 
including the selected effective macroblock has been detected 
due to transmission noise or the like and the region is not a 
moving object. 

30 For example, in the case of the example shown in Fig. 2, 
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when the predetermined threshold value is 5, the moving object 
determination unit 104 determines only the region of the group 
of the effective macroblocks 304 to 327 as the one contained 
in the region of a moving object, that is, determines that the 
5 region of the group of the effective macroblocks 304 to 327 is 
a moving object- For the sake of simplicity, in the example 
shown in Fig. 2, only the group of adj acent effective macroblocks 
304 to 327 is determined as a moving object. Of course, there 
is a possibility that two or more groups of adjacent effective 
10 macroblocks are determined as moving obj ects . The moving obj ect 
determination unit 104 thus repeatedly performs the process of 
determiningmoving obj ects, as shown in Fig, 3, until the counting 

* for each of all the adj acent effective macroblocks on the screen 

in 

is carried out. 

^ 15 As mentioned above, in accordance with the first embodiment 

of the present invention, the moving object detector analyzes 
an incoming coded stream so as to extract a motion vector and 
other encoding parameters for each macroblock from the coded 
stream, identifies one or more effective macroblocks based on 

20 the acquired motion vector and other encoding parameters, and 
then determines whether or not there is one or more moving obj ects 
by determining whether each identified effective macroblock 
forms a group of effective macroblocks whose number is equal 
to or greater than a predetermined threshold value in cooperation 

25 withothereffecti-vemacroblocks which are directly or indirectly 
adjacent to each effective macroblock in question. Accordingly, 
the moving object detector can detect one or more moving objects 
with a high degree of accuracy even when an encoding mode that 
does not generate a motion vector is selected for encoding each 

30 macroblock. The moving object detector can also neglect 
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macroblocks with a large motion vector which are spread on the 
screen due to transmission noise or the like, so that the moving 
object detector can detect one or more moving objects at a high 
speed and with a high degree of accuracy. 
5 The moving object region determination method as shown 

in Fig. 3 is only an example. Any method of determining one 
or more moving object regions, limited only by use of effective 
macroblocks, can be adopted. 

In accordance with the first embodiment mentioned above, 

10 as shown in Fig. 1, the coded stream analyzer 102 is so constructed 
as to receive and analyze a coded stream from a moving picture 
encoder (not shown in the figure) so as to extract encoding 
parameters from the coded stream. However, the present 
embodiment is not limited to the case. In a variant of the first 

15 embodiment of the present invention, a moving object detector 

301 shown in Fig. 4 can be so constructed as to receive encoding 
parameters directly from a moving picture encoder (not shown 
in the figure) by way of an encoding parameter input circuit 

302 so as to acquire the encoding parameters. In other words, 
20 the moving object detector 101 shown in Fig. 1 is an example 

of moving object detectors disposed on a side of a moving image 
decoder, which receives a coded stream from a moving picture 
encoder by way of a transmission path (not shown in the figure) . 
In contrast, the moving object detector 301 shown in Fig. 4 is 
25 an example of moving object detectors disposed on a side of a 
moving image decoder, which inputs encoding parameters used when 
a moving image encoder encodes a moving image directly from the 
moving image encoder . 

30 Embodiment 2. 
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Fig . 5 is a diagram showing the structure of a moving ob j ect 
detector 401 according to a second embodiment of the present 
invention. In Fig. 5, reference numeral 402 denotes a moving 
object coordinate storage unit for storing the on-screen 
5 coordinates of each effective macroblock which has been 
determined as a macroblock contained in a moving object by a 
moving object determination unit 104, reference numeral 403 
denotes a motion vector storage unit for storing a motion vector 
accompanied by each effective macroblock which has been 

10 determined as a macroblock contained in a moving object by the 
moving object determination unit 104, and reference numeral 404 
denotes a post-movement coordinate calculation unit for 
calculating the position of each moving object which has moved 
from the on-screen coordinates of macroblocks included in each 

15 moving object, which are stored in the moving obj ect coordinates 
storage unit 402, and the corresponding motion vectors stored 
in the motion vector storage unit 403. The same reference 
numerals as shown in Fig. 1 refer to the same components as those 
of the moving object detector of the first embodiment shown in 

20 Fig. 1, respectively, and therefore the explanation of those 
components will be omitted hereafter. 

In the following, an operation specific to the moving 
object detector of the second embodiment will be explained as 
compared with the operation of the moving object detector of 

25 the first embodiment. 

When the moving object determination unit 104 determines 
that there is one or more moving objects, or there is an effective 
macroblock contained in a moving obj ect, like that of the moving 
object detector of the first embodiment, in the moving object 

30 detector according to the second embodiment, the moving object 
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coordinate storage unit 402 stores the on-screen coordinates 
of each effective macroblock which has been determined as a 
laacroblock contained in a moving object by the moving object 
determination unit 104 . Here, the on-screen coordinates of each 
5 effective macroblock can be any ones, limited only by a common 
definition provided for each effective macroblock. For example, 
the on-screen coordinates of each effective macroblock can be 
the number of pixels in the horizontal direction and the number 
of pixels in the vertical direction, both of which are counted 

10 from the upper left corner of the screen to a pixel at the center 
of each effective macroblock. 

The motion vector storage unit 403 simultaneously stores 
a motion vector accompanied by each effective macroblock which 
has been determined as a macroblock contained in a moving object 

15 by the moving object determination unit 104. 

The post-movement coordinates calculation unit 404 then 
calculates a prediction of the on-screen coordinates of each 
effective macroblock in the next frame which has been determined 
as a macroblock contained in amoving ob j act based on the on-screen 

20 coordinates of each effective macroblock which has been 
determined as a macroblock contained in the moving object, which 
are stored in the moving object coordinates storage unit 402, 
and the corresponding motion vector stored in the motion vector 
storage unit 403, and outputs the calculated prediction to the 

25 moving object determination unit 104. 

There is a method of calculating a prediction of the 
on-screen coordinates of each effective macroblock in the next 
frame, which the post-movement coordinates calculation unit 404 
employs, by adding the corresponding motion vector stored in 

30 the motion vector storage unit 403 to the on-screen coordinates 
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of each effective macroblock in the current frame which has been 
determined as a macroblock contained in a moving object, the 
on-screen coordinates of eachef f ectivemacroblock in the current 
frame being stored in the moving object coordinates storage unit 
402. There can be another method of calculating a prediction 
of the on-screen coordinates of each effective macroblock in 
the next frame by adding a mean value of the motion vectors of 
effective macroblocks including each effective macroblock in 
qxiestion, which have been determined as macroblocks contained 
in a moving ob j ect , to the on-screen coordinates of each effective 
macroblock in the current frame . In other words, the prediction 
of the on-screen coordinates of each effective macroblock in 
the next frame can be determined by any technique, limited only 
by the use of the on-screen coordinates of each effective 
macroblock in the current frame which has been determined as 
a macroblock contained in a moving ob j ect, and the corresponding 
motion vector. 

The on-screen coordinates thus calculated become a 
powerful candidate for the position of a moving object in the 
next frame as long as the moving object is not moving very fast 
with a sudden movement. In other words, there is a possibility 
that a macroblock in the current frame which has the post-movement 
coordinates of an effective macroblock in the preceding frame, 
which are acquired by the post-movement coordinates calculation 
unit 404, is an effective macroblock contained in amoving obj ect . 

Thus, the moving object determination unit 104 according 
to the second embodiment determines whether there is one or more 
moving objects based on effective macroblocks having on-screen 
coordinates predicted by the post-movement coordinates 
calculation unit 404 and effective macroblocks identified by 
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the effective macroblock identification unit 103, like that of 
the first eitibodiment . 

For example, the moving object determination unit 104 can 
implement a logical AND operation on effective macroblocks having 
5 on-screen coordinates predicted by the post-movement 
coordinates calculation unit 404 and effective macroblocks 
identified by the effective macroblock identification unit 103, 
and can perform a moving object determination process shown in 
Fig. 3 on only effective macroblocks acquired by the logical 

10 AND operation, like that of the first embodiment. As a result, 
the niomber of groups of effective macroblocks which have to be 
processed in the moving object determination process shown in 
Fig. 3 can be reduced, and therefore the moving object 
determination process can be speeded up. 

15 Concretely, as shown in Fig. 6, the moving object 

determination unit 104 performs the moving obj ect determination 
process as shown in Fig. 3 on only effective macroblocks 402 
to 416, which correspond to a logical AND operation result of 
a prediction of the on-screen coordinates of effective 

20 macroblocks 304' to 327' estimated from effective macroblocks 
304 to 327 in the immediately preceding frame shown in Fig. 2, 
for example, and the on-screen coordinates of effective 
macroblocks 402 to 425 in the current frame, which are identified 
by the effective macroblock identification unit 103, without 

25 performing the moving object determination process as shown in 
Fig. 3 on effective macroblocks 401 and 426 to 428, thus speeding 
up the moving object determination process. As a result, 
according to this moving object determination process, the 
macroblocks 402 to 425 are counted as effective macroblocks which 

30 are directly or indirectly adjacent to each of them. 
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As an alternative, after the moving object determination 
unit 104 implements a logical AND operation on effective 
macroblocks having on-screen coordinates predicted by the 
post-movement coordinates calculation unit 404 and effective 
5 macroblocks identified by the effective macroblock 
identification unit 103, the moving object determination unit 
104 can perform the moving object determination process in 
consideration with the movement of each effective macroblock 
in the immediately preceding frame, thereby improving the 

10 accuracy of determination of moving objects. 

Concretely, as shown in Fig. 6, the moving object 
determination unit 104 performs the moving object determination 
process as shown in Fig. 3 on only the effective macroblocks 
402 to 416, which correspond to the logical AND operation result 

15 of the prediction of the on-screen coordinates of the effective 
macroblocks 304 ' to 327' estimatedf romthe ef f ectivemacroblocks 
304 to 327 in the immediately preceding frame shown in Fig. 2, 
for example, and the on-screen coordinates of the effective 
macroblocks 402 to 425 in the current frame, which are identified 

20 by the effective macroblock identification unit 103, so that 
the moving object determination unit 104 counts the number of 
effective macroblocks 402 to 416 which are directly or indirectly 
adjacent to each of them, thus improving the accuracy of 
determination of moving obj ects . As a result, according to this 

25 moving object determination process, only the macroblocks 402 
to 416 are counted as effective macroblocks which are directly 
or indirectly adjacent to each of them. 

As mentioned above, in accordance with the second 
embodiment of the present invention, the moving object detector 

30 identifies one or more effective macroblocks and then determines 
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whether or not there is one or more moving objects by determining 
whether each identified effective macroblock forms a group of 
effective macroblocks whose number is equal to or greater than 
a predetermined threshold value in cooperation with other 
5 effective macroblocks which are directly or indirectly adjacent 
to each effective macroblock in question, like that of the 
above-mentioned first embodiment. Accordingly, the moving 
object detector can detect one or more moving objects with a 
high degree of accuracy even when an encoding mode that does 

10 not generate a motion vector is selected for encoding each 
macroblock. The moving object detector can also neglect 
macroblocks with a large motion vector which are spread on the 
screen due to transmission noise or the like, so that the moving 
object detector can detect one or more moving objects at a high 

15 speed and with a high degree of accuracy. 

Furthermore, in accordance with the second embodiment, 
the moving object determination unit 104 determines whether or 
not there is one or more moving objects based on a prediction 
of the on-screen coordinates of each effective macroblock in 

20 the immediately preceding frame which has been determined as 
a macroblock contained in a moving object, and the on-screen 
coordinates of each effective macroblock in the current frame 
identified by the effective macroblock identification unit 103 . 
Accordingly, the moving object determination process can be 

25 carried out at a high speed and with a high degree of accuracy. 

In a variant of the second embodiment, instead of 
extracting encoding parameter by analyzing the incoming coded 
stream, a moving object detector 501 shown in Fig. 7 receives 
encoding parameters directly from a moving image encoding unit 

30 (not shown in the figure) by way of an encoding parameter input 
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circuit 502. In other words, the moving object detector 401 
shown in Fig. 5 is an example of moving object detectors disposed 
on a side of a moving image decoder, which receives a coded stream 
from a moving picture encoder by way of a transmission path (not 
5 shown in the figure) . In contrast, the moving object detector 
501 shown in Fig. 7 is an example of moving object detectors 
disposed on a side of a moving image decoder, which inputs encoding 
parameters used when a moving image encoder encodes a moving 
image directly from the moving image encoder. 
10 Furthermore, in the above-mentioned second embodiment, 

a prediction of the on-screen coordinates of each effective 
^ macroblock in the immediately preceding frame that has been 

;t determined as a macroblock contained in a moving obj ect, as shown 

g in Figs. 5 and 7, is input to the moving object determination 

15 unit 104 from the post-movement coordinate calculation unit 
effective macroblock from 404. The present invention is not 
limited to the case. As an alternative, the moving object 
determination unit 104 can receive the prediction of the 

'11 

on-screen coordinates of each effective macroblock predicted 
20 by the post-movement coordinates calculation unit 4 04 so as to 
implement a logical AND operation on effective macroblocks having 
on-screen coordinates predicted by the post-movement 
coordinates calculation unit 404 and effective macroblocks 
identified by the effective macroblock identification unit 103, 
25 and then output a- logical AND operation result to the moving 
object determination unit 104. 
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Embodiment 3. 

Fig. 8 is a block diagram showing the structure of an image 
monitoring system according to a third embodiment of the present 
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invention. In the figure, reference numeral 601 denotes a camera 
for capturing an image of a monitoring site, reference numeral 
602 denotes a moving image encoding unit for encoding an image 
captured by the camera 601, reference numeral 603 denotes a 
5 transmission path, such as the Internet, via which a coded stream 
encoded by the moving image encoding unit 602 is transmitted 
to a monitor room, reference numeral 604 denotes a moving object 
detector according to any one of the above-mentioned first and 
second embodiments, reference numeral 605 denotes a moving image 

10 decoding unit for decoding a coded stream received by way of 
the transmission path 603, reference numeral 606 denotes a 
monitor for displaying an image decoded by the moving image 
decoding unit 605, reference numeral 607 denotes a warning unit 
for generating a warning sound, warning light, a warning signal, 

15 or the like when one or more moving objects are detected by the 
moving object detector 604, and reference numeral 608 denotes 
a storage unit for storing coded streams. 

In operation, the camera 601 is fixedly or movably directed 
to the monitoring site and always captures an image of the 

20 monitoring site so as to supply the image to the moving image 
encoding unit 602. 

The moving image encoding unit 602 encodes the moving image 
supplied from the camera 601 and transmits a coded stream by 
way of the transmission path 603. The coded stream transmitted 

25 by way of transmi-ssion path 603 is input to the moving object 
detector 604, the moving image decoding unit 605, and the storage 
unit 608 disposed on a receiver side, i.e., a decoder side. 

The moving object detector 604 then analyzes the input 
coded stream so as to determine whether or not there is one or 

30 more moving objects in the moving image, and outputs a 



26 



determination result to the warning unit 607. In other words, 

since the moving object detector 604 needs to analyze the coded 

stream in the case of the image monitoring system shown in Fig. 

8, the moving object detector 604 can be either a moving object 
5 detector 101 according to the first embodiment shown in Fig. 

1 or a moving object detector 401 according to the second 

embodiment shown in Fig. 5. 

Then, the warning unit 607 emits a warning sound, warning 

light, a warning signal, or the like so as to notify an observer 
10 of the detection of moving obj ects when one or more moving objects 

have been detected by the moving object detector 604. 

The moving image decoding unit 605 then decodes the coded 

stream transmitted thereto by way of the transmission path 603, 

and outputs the decoded stream to the monitor 606 as an image 
15 signal . The monitor 60 6 displays the corresponding moving image 

so as to allow the observer to keep a watch on the monitoring 

site . 

The storage unit 608 stores the coded stream at all times 
so as to allow reproduction of the corresponding moving image 
20 at anytime by the monitor 606 or the like. 

Therefore, in accordance with the third embodiment, the 
observer can be allowed to discover an intruder to the monitoring 
site by observing the monitor 606 only when the observer is 
notified of detection of the intruder from the warning unit 607 
25 without having to- keep a watch on the monitor 606 at anytime, 
and therefore the load imposed on the observer can be reduced 
significantly. 

Furthermore, as shown in Fig. 9, when the moving object 
detector 604 discovers an intruder or the like as a moving obj ect , 
30 the moving object detector 604 can send a signal indicating a 
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request for intra coding to the moving image encoding unit 602 
byway of the transmission path 603, and the moving image encoding 
unit 602 can intra-code following frames when receiving the intra 
coding request signal. Thus, when an intruder or the like is 
discovered by the moving object detector 604, a coded stream, 
which starts from an intra-coded frame, can be stored in the 
storage unit 701, and it is therefore easy to extract only the 
frame at which an intruder or the like is discovered. 

Furthermore, instead of or in addition to transmitting 
the intra coding request signal to the moving image encoding 
unit 602, it is also possible to transmit a signal indicating 
a request for a change in an encoding parameter, such as a 
quantization step, and then store an image with a lower 
compression rate than usual, i.e., a higher resolution and higher 
definition in the storage unit 701 when a moving object such 
as an intruder is discovered. Thus, instead of or in addition 
to extraction of the frame at which a moving object such as an 
intruder is discovered, an image with a high compression rate 
is stored when any moving object, such as an intruder, is not 
discovered, whereas when a moving object, such as an intruder, 
is discovered, the moving object can be confirmed with a clear 
image with a low compression rate. 

Furthermore, as shown in Fig. 9, at the same time that 
a coded stream which starts from an intra-coded frame is stored 
when one or more moving objects are detected, the moving object 
detector 604 can output information such as a time and 
characteristics of the image to the storage unit 701 as metadata 
so as to store the metadata in the storage unit 701 together 
with the coded stream which starts from the intra-coded frame. 
Accordingly, when a coded stream which starts from an intra-coded 
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frame is thus recorded together with its metadata, the present 
embodiment offers an advantage of being able to easily locate 
the start of the recorded sequence, for example, when reproducing 
a corresponding moving image when a moving object, such as an 
intruder, is discovered, and hence to easily search for the frame 
at which the moving object is discovered, by referring to the 
metadata stored in the storage unit . Metadata canbe transmitted 
from the moving image encoding unit 602 other than the moving 
object detector 604, by way of the transmission path 603, to 
the storage unit 701. 

The storage unit 701 does not have to store the coded stream 
from the moving image encoding unit 602 at all times, but can 
alternatively store the coded stream from the moving image 
encoding unit 602 in response to the metadata from the moving 
obj ect detector 604 or the moving image encoding unit 602 . Since 
the coded stream is thus stored in the storage unit 701 after 
one or more moving objects have been detected or only while one 
or more moving objects are detected, the amount of storage used 
in the storage unit 701 can be reduced compared with that provided 
by prior art techniques and therefore exhaustion of storage such 
as a tape can be prevented. 

Embodiment 4 . 

Fig. 10 is a block diagram showing the structure of an 
image monitoring system according to a fourth embodiment of the 
present invention. In the figure, reference numeral 801 denotes 
a moving object detector according to any one of the first and 
second embodiments of the present invention, and reference 
numeral 8 02 denotes a moving picture encoding unit that operates 
based on a captured image from a camera 601 and a detection result 
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from a moving object detector 801. The other structure of the 
image monitoring system according to the fourth embodiment of 
the present invention is the same as that of the above-mentioned 
third embodiment as shown in Figs . 8 and 9 and the same reference 
5 numerals refer to the same components, and therefore the 
explanation of those components will be omitted hereafter. 

In operation, the camera 601 is fixedly or movably directed 
to a monitoring site and always captures an image of the monitoring 
site so as to supply the image to a moving image encoding unit 

10 802 . The moving image encoding unit 802 encodes the moving image 
supplied from the camera 601, and extracts encoding parameters 
from a coded stream and then transmits them to the moving object 
detector 8 01. The moving image encoding unit 8 02 also transmits 
the encoding parameters to a moving image decoding unit 605 by 

15 way of a transmission path 603. 

The moving object detector 8 01 then analyzes the input 
encoding parameters so as to determine whether or not there is 
one or more moving objects in the moving image, like that of 
either of the first and second embodiments, and outputs a 

20 determination result to the moving image encoding unit 802. In 
other words, since the moving object detector 801 is disposed 
on the encoder side and does not need to analyze the coded stream 
in the case of the image monitoring system shown in Fig. 10, 
the moving object detector 8 02 can be either a moving object 

25 detector 301 acco-rding to the first embodiment shown in Fig. 
4 or a moving object detector 501 according to the second 
embodiment shown in Fig. 7. 

When the moving image encoding unit 8 02 receives the 
determination result indicating detection of moving obj ects from 

30 the moving object detector 801, the moving image encoding unit 
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802 transmits a warning signal indicating the detection of moving 
objects to the moving image decoding unit 605 by way of the 
transmission path 603. 

The moving image decoding unit 605 then decodes the coded 
5 stream transmitted thereto by way of the transmission path 603, 
and outputs the decoded stream to a monitor 606 so as to reproduce 
the captured image. A warning unit 607 emits a warning sound, 
warning light, a warning signal, or the like based on the warning 
signal from the moving image encoding unit 8 02. 

10 Therefore, in accordance with the fourth embodiment, an 

observer can be allowed to discover an intruder to the monitoring 
site by observing the monitor 60 6 only when the observer is 
notified of detection of the intruder from the warning unit 607 
without having to keep a watch on the monitor 60 6 at anytime, 

15 and therefore the load imposed on the observer can be reduced 
significantly. 

Since the moving image encoding unit 802 of the fourth 
embodiment is so constructed as not to transmit the coded stream 
to the moving image decoding unit 605 by way of the transmission 

20 path 603 when the moving object detector 801 determines that 
there is no moving obj ect at the monitoring site, the coded stream 
flows through the transmission path 603 only when one or more 
moving objects, such as an intruder, are detected and therefore 
the load imposed on the transmission path 603 can be reduced. 

25 Furthermore, even in the fourth embodiment , when the moving 

object detector 801 determines that there is one or more moving 
objects at the monitoring site, the moving image encoding unit 
802 can intra-code following frames, and a storage unit can 
therefore store a coded stream, which starts from an intra-coded 

30 frame, when one or more moving objects are discovered at the 



31 

monitoring site. 

Furthermore, at the same time that a coded stream which 
starts from an intra-coded frame is stored when one or more moving 
objects are discovered at the monitoring site, the moving obj ect 
5 detector 801 can output information such as a time and 
characteristics of the image to the storage unit 701 as metadata 
so as to store the metadata in the storage unit 701 together 
with the coded stream which starts from the intra-coded frame. 
Accordingly, as previously explained in the third embodiment, 

10 when the moving object detector 801 detects an intruder at the 
monitoring site, a coded stream which starts from an intra-coded 
frame is stored in the storage unit 701 together with its metadata, 
and therefore an observer can easily extract only the frame at 
which the intruder is discovered, by referring to the metadata 

15 stored in the storage unit. When the moving object detector 
801 determines that there is one or more moving objects at the 
monitoring site, the moving image encoding unit 8 02 can perform 
encoding with a lower compression rate than usual, i.e., a higher 
resolution and higher definition on the image by changing an 

20 encoding parameter, such as a quantization step, prior to the 
discovery of the one or more moving objects so as to generate 
a coded stream and output it to the decoder side by way of the 
transmission path 603 . Therefore any intruder can be confirmed 
with a clear image while the observer can easily extract only 

25 the frame at which the intruder is discovered, by referring to 
the metadata stored in the storage unit. 

Metadata can be transmitted from the moving image encoding 
unit 802 other than the moving object detector 801, by way of 
the transmission path 603, to the storage unit 701. In this 

30 case, since the storage unit 701 can store the coded stream from 
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themoving image encodingunit 802 while one or moremoving objects 
are detected, the amount of storage used in the storage unit 
701 can be reduced compared with that provided by prior art 
techniques. In addition, when a coded stream which starts from 
an intra-coded frame is recorded together with its metadata, 
the present embodiment offers an advantage of, when reproducing 
a corresponding moving image when a moving object, such as an 
intruder, is discovered, being able to easily search for the 
frame at which the moving object is discovered, and to easily 
locate the start of the recorded sequence. 

Many widely different embodiments of the present invention 
may be constructed without departing from the spirit and scope 
of the present invention. It should be understood that the 
present invention is not limited to the specific embodiments 
described in the specification, except as defined in the appended 
claims . 



